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(54) Abstract Title: Image stabilisation using field of view and image analysis. 

(57) In a video image stabilisation system 20 for removing unwanted camera shake, processing device 50 * 
receives signals indicating both changes in the camera field of view (FOV) as well as plural captured 
images, for example first and second images; the field of view may be adjusted by camera panning, tilt or 
focal changes. An image stabilisation adjustment is then performed using both the FOV changes, i.e. 
intending camera movements or focus changes, and by analysis of the images themselves. Thus, the 
system discriminates between intended, wanted image changes caused by camera movements such as 
panning or tilting, and unintended image changes due to camera shake. The final image portions (84, 94: 
Figure 2) displayed are adjusted to remove shake. Stabilisation to remove unwanted, inadvertent camera 
movement may include image scaling and/or generating a histogram of image differences. 
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BACKOROT TNTn nz: THF rm^r-r^ 
1. Held of the Invention. ~ 

fOOOJJ TTie present invention relates to a video camera stabilization sv Stem „ 
particularly, to a ^ ^ g and, more 

image motion. ^ ' miWanted Md ^ar 

2. Description of the Related Ait 

(0002) There are numerous known video surveillance systems which may be used to rr^ 
moving object such as a person or vehicle So™, u . may oe used to track < 

a stationary field of vrew^ r ^ ^ ^ 8 ** ^ *** 

nary neid vrew (FOV). To folly cover a given surveillance site with a fixed Cam ~. 

cameras. 

[0003] Movable cameras which mavrkiiTi +;n - * t 

_ may pan * faIt ««« ^ may also be used to track 

of the vrdeo feeds and system mtegratjon hardware such as movers and switchers 
associated therewith. Control agraals for dire^inc the ™„ 

^ • . e arrecting the pan, nit, zoom movements tyoicallv 

originate from a human operator via a iovsrieV ^ ly 

.00041 a u] peratOT a J^nck or from an automated video tracking system. 

controls^ thecamer* car, also ^unintentionally moved by externa! forces suchTair 
movement or mounting arm vrbradon. The inadvertent camera 

fOOOS, What is needed in the art is a video system capable of sumilizing an image to 
compensate for inadvertent movement of the camera. 

SUMMARY" nv rm? t ^TSNTiqn 

-M *e ta* upM1 fc ^ ^ „ „ fieM of ^ • - 

•ralKBoftroofthe caponed images. 
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10007) HninviWoiconpisw, in one Cum a~~,e. ^ . 

, m ™ ,. • u 8 ^ nns ^" M ^8«"e<lto capture a video 

am*, where,, 4, h, , Kleelive , *™» «*° 

-Woe is operebly coupled » fi. ^ ^ 8 

M-h. of «* field of view ™T!r * ^ 

^ MltecMMraml '"MSe!eapiuredbyihecanen The 

uuage « d,e seooud ta,a e c o*ed upouth. Wfc^ „, „„, ' *" 
a~ly S i.ofure &! , aB dreco» 1 ire ages . ^ " ^ 2) » 

^^^^dere^ed^^pre^^^ ^ 

locaoouoffteaerecreddiapl.yp^^^^,,^^ 

IOOO»1 ^ -«»^c<^ iu^ta^,^ 

"^■^P^ofiree^c^^d.^^,^ ^ 

•-r-*'**.-— IhcpreccesregdeWcereceivc 
^^oaec^P^^^^ ^ 

^byadj™^^^^^^^ 

captured image and a previously captured image. 
[0010, J^tionco^^ 

™>ge. Tbe method includes capturing a phnality of images with a video camera, the video 
camera bemg mo„ at ed on a stationary support and having an adjustable field of view, and 
^Sdreneldofviewofme camera, ^methodalsoincmdes gra^g a t^t image 

bcmg ]esS than the entire captured image and determining a stabfl tog adjustment for * e 



within the second image. 

fOOU, ^^ofthefirrtands^^ 

andsecon ^^^^^..^^ fc-iiIl(bD-h|(| ^ 
ana yas of the first and second images includes aligning one of me first and second images 
wuh the other of the first and seeond images based upon signal, indicative of the field of 
-wreceive^ 

*e ahgned unages indicate the occurrence of unmtentiona! camera movement An image 
Afference of the aligned images is used to determine if the aligned images indicate the 
ocmrence of unintentional camera movement A histogram of the image difference of the 
ahgned unages is generated and determining whether unintentional camera movement has 
occurred is accomplished by identifying a maximum peak in the histogram and a location of 
the maximum peak. When the maximum peak is not substantia.* centered on zero 
ummennonal motion is determined to have occurred and when such unintentional motion is 
detected, determination of the stabilizing adjustment includes identifying adjustments that 
minimize the image difference. 

{0012J The invention comprise, m ^ another form thereof, a method of stabilizing a 
vtdeo unage. The method includes intentionally varying a field of view of a video camera by 
use of control signals, capturing a plurality of images with the camera, each of the images 
being associated with a field of view, displaying a selected portion of each of the plurality of 
unages and adjusting the selected portion that is displayed for at least one of the plurality of 
.mages. The adjusting step compensates for inadvertent movement of the video camera and 
,s based upon an intended field of view of the camera and an analysis of the at least one of the 
plurality of images and a second one of the images. 

10013] An advantage of the present invention is that it provides video unages which are 
stabmzed to limit or eliminate the distracting and unpleasant effects caused by inadvertent 
movements of the camera. 

f0014J Another advantage of the present invention is that it allows for the stabilization of 
unages acquired while the camera is in motion. Thus, the present invention can compensate 
for unmtentional movement of a pan, tilt, zoom camera, such as that caused by s^rppon arm 



vibration or the effect of wind on the camera housing even while A, , 

purposely change its field of view. ^ * » d 

understoodbyreWetothefonowingd^ 
» conjunct™ with the accompanying drawing,, wherein: 

Figure 1 is a schematic view of a vtrW «^ M ;n- 
present invention. " ^ * ™* 

Figure 1 and the resulting images captured by the camera. 

Figure 3 is a schematic view of a portion of the captured images of Figure 2 
Frgure 4 is a block diagram of the processing device of Figure 1 
Figure 5 is a flowchart representing one embodiment of the operation of the video 
surveillance system of Figure 1. 

Figure 6 is a histogram used in the analysis of two images. 

fOOlol Corresponding reference characters inHiVnf,- ™,~>„ 

, . "**raciers indicate corresponding parts throughout the 

be construed a$ limiting the scope of the invention to the precise fimn disclosed. 

DESCRIPTrnM w w T wssmrr lyr yp^^ 

Figure 1 . 20 mCudes a camera 22 which is located within a partially spherical 

e n closu re2 4and ra ountedon S u PP ort25. Stationary support 25 may take rr^y forms, ^ 
- an outward* extending suppo* arm extending mom an exterior edge of a building which 
may subject *e supported camera to unintentional movement resulting from wind, vibrations 
generated by the camera motors, nearby machinery or a myriad of other sources. Endosure 

and sunulteneously prevent individuals in the environment being observed by camera 22 
^ detennmmg the orientation of camera 22. Camem^ in cludes a control, er and motors 
winch prov.de for the panning, tilting and adjustment of the foca, length of camera 22 
Pamtmg movement of camera 22 is represented by arrow 26, tilting movement of camera 22 
* presented by arrow 2* and the changing of the focal length of the lens 23 of camera 22 



».e., zooming, i s represented by airow 30 .u , 

m . . y • AS Shown WIlh reference to coordinate system 2 1 

panning modon may track movement along the x axis tilnW™* res ^21, 

Camera Systems brand camera system, such as the G 3 Basic AutoDome « camera and 
enclosnre, which are ava„ab,e foorn Bosch Security Systems, (be. formerly^ 
Commumcation, Securiry & Imaging, Inc. having a place of business in Lancaster 

m U.S. Pat No. 5,627,616 entitled Surveillance Camera Svsten, fc • fc . 
he*in by reference. ^ * ^ 

[0018, Sys« e m20alsoinclu deS aheadendunit32. Head end unit 32 may include a video 
switcher or a video multiplexer 33 For examole ^w, * . ■ . 60 
v , -. . t P eX8mpIe ' me head end "»t may include an Allegiant 

Z^^!^ i T fcl ^^ , ^ H « d ^32 include^ 
^34^^36 for operator input Head end unit 32 also includes a display 
dev.ee „ the form of a monitor 38 for viewing by the operator. A 24 volt a/c power source 
40 is provded to power both camera 22 and a processing device 50 mat is operably coupled 
to both camera 22 and head end unit 32. 
[0019, ^^^20^^^^ 

e.ther senary or moveab,e cameras or some combination thereof to provide coverage of a 
fcrger or more convex surveillance area. One or more VCRs or other form of analog or 
4gdd recording device may also be connected to head end unit 32 to provide for the 
recordmg of the video images captured by camera 22 and other cameras in the system 
[0020, n,e general concepts underlying the operation of an inrage stabilization system in 
accordance with the present invention will now be discussed win, reference to Figures 2 and 
3. Camera 22 includes an image-capturing device such as a charge coupled device-(CCD) 
that acquires a four-sided CCD video image 80 defined by CCD image boundary &"> 
Processing device 50 identifies or selects a central portion of CCD image 80 to be displayed 
on a screen of monitor 38 for viewing by an operator of system 20. More particular* 
processing device 50 identifies a display image 84 defined by display image boundary' 86 for 
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viewing on the screen of monitor 38. In one embodmw a . . 
video image, ic display im a&e * ■ , T he S « J ^ Portion of the CCD 

CCD v,W ' ^ between 60% and 90% of the 

la A Una ^ e ' a P 16 ^ 116 ^ embodiment, CCD image 80 can be approximately 30^ 

preferred embodiment, approximately 23% of cm ; 0n ^ * 

o^nitor38.HoweverleaseofClTcr 

osc oi mustration, CCD image 80 is shown in Fistwe Iml, 

image that is smaller than the CCD im,™ . § y ublizin e a 

erttu.nmeCCD.mage.pnKJessjngdeviceSOcanadjustthepositionsof 
the boundanes defining the display image relative to the entire CCD mL f 
acmiiT^H - TO ucu image for sequentially 

ZLrTr^ 88 ' Fi ^ 2 "^-» 

i.e., inadvertent, movements of camera 22 The mt^^A ^ 

I ile tended movement of camera 22 that nm^ 

movement or mounting arm vibration. 

™ 8 e Kim, « Wve „ ^ ^ , 2 ^ * 

W Figures 2 and 3 sC^dcUy TOo ^ <#1 , 1 , 11-1 



m. i « Z C0 ™P« 1 ^ » J*™i°S move™™ of cam„a 22) and venieal 

- -mm, diction, oh™*, h „ ot(amc „ • «• 

4T »y b. «• ,o b= ^ta, to 4, difW. b,,™. ^ ^ ^ V - 

kmm* h2 „d u ^ fomm „ ^ comspmdiiig ^ 

image boondmos 82, », respectively. Thai is, ap = | _ ,,3 1 
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I0OMJ Ti =i>««ni>toltoofd 1 ev i ,| U B <)faTand4 p. 

^ea in a wade vanety of computer vision and video processing 
applications. Two well-Jcnnum wj,^ processing 
and feature * ,V 1 » «&* correlation-based tenmlate matching 

96 m each of the two images might be accomplished using a feanm, ma ^- 
«0"W^ airf mmched in ihe two sepanie iongts. 

mm Although such , We taaScWc* eppcC M ta M|)1<>yed Wfc 

co^upp^ A sunt of square differences approach typically relies upon the 
follovwngniatheiijaticalrelatioiisliip: 

£m *$ti">"Qh.y t f. (i) 

where 2T is the total error, / is the image. M is the model template, v) are the mode, 

<h**ence between y and y corresponds to AT. The relation between foe transformed image 
coordinates and the model image coordinates i***fi~~iu g£ 
mM . Ko J j 8 ^ ecooi *nate Sls defined by a mouon model. Different motion 
models can be used and an affine transformation as nm**.. k„i 
, . uimon as presented below models rotation 

shearing, and translation: 



0 0 




(2) 



Ws motion model (2) mounts a leas, 3 ^ K b ^ 

^|^*on, the illustrated embodiment utilizes a normalized cross-correlation abroach 
instead. Template matching using cross-correlation « 

distances: ^^lahon is based upon squared Buclidean 



(3) 



Wherein/is the image, e.g. second CCD image 88, and the summation is over, y under ft 
expansion of J provides: 



<<"- v )-Zl^'(».J.)-2/(a i) .),(,-„, J ,- v)+ ^ ;[ . lliJ ,_ ) . ) ] (4) 

cross-correlation term: ^ ung 

ft- - several disndvanrngen. however, m ^ te ^ 
^^'^-^oi.nHsr^m.m*, IVe.^ooanca™ J y bc' 
obtained by normalizing the image and feature vectors to unit length, providing: 

r(u,v) = 

referred to as normahzed cross-correlation. By maximizing the .ormalizcd correlation 



(6) 



determine a value for AT and AP. 

fOWl] In addition to determining the actual change in ftefeu** ■ 

nnri a d . mange in tfte field of view as indicated bv at 

i» - M u camera 22 commumcares to processW device 

8ny» in the pan, tilt and zoom positions of camera 22 and A * * ' 

of AT, and AP,. * ^ 

10033) The intended change m the FOV can also be dei^r^wii. 

FftrilM „ . . , ^ aiso oe determined by alternative methods 

^by^^^^^ Alternatively the camera eount^ 

other data gathered fiom system 20. i?- °y analyzing 

[0034] Once both the actual change in the FOV of the r»m»« 

A p an , t . 1JL 6 camera, as represented by AT and 

AP, and the intended change in the FOV nf rk» 

hOV of die camera, as represented by ATj and APi are 

values ATg and APg represent the difference between dip armai u 

Am , « . *«cuuc ocween the actual change in the FOV (AT and 



50 compensa^ for such inadvertent movement of camera 22 bv «r ■ . 

In Figure 2 an adiu«»H „• , * ^ ^ Values °^T E and AP E 

'igure 2, an adjusted display nnage 98 defined by adjusted disnW k , 

displayed on the screen of monitor 38. ^ 100 WOU]d be 

[0035] When communicating images to head end unit 32 for dL<n )a v 

boundaries 86, 100. 7 botadan «. e '8> display 

(0036] As long as the entire adjusted disolav fan.*, o* • 

me CCD v,deo image are separated by mar gi n s on all four side, n« „• 
display imag e is, thus, limited to width of thXesnond ° f *" 

which the display imace can be 7 ^ That is, the extent to 

we uz^piay image can be repositioned within the rrn - . 

camera. encountered by the surveillance 

V ynnagewmie the camera is stationary, but also provides fbrfe^r - , 



address such a change in die zoom setting of cam*™ 9-> • , 

■*» ^ by prec*^ device 50 ^ fc J™""*" *** 

on may not appreciably degrade the value of (he hnaee aaMlb_- 
Moreover, when ft. to, tojft b ^ _ ^^JT. ^^"^ 
.«w^^ cnfawMyW f ^^^^^dore s 

"^^wfll^be^dtoacdveaswlJ* ^emoml tag, 

(0039J It is also possible to continue to perform th* - 

aT^ ^s e^jeets depicted in each of die images are depicted al a common scale and cai^be^ 
aw ^-PKebft^eometrfctrens^re.,,^ 

^^ft^fLrjr^^L 

■»»fi»mm i .ai i ,s,i lm 8o ma „ sni , wiailswlid * 

No. Al entided Modon-Bared Tree.™ Wift^T' I ^' 

l^byiaco^tedh^inoyrefereno. -^Tnt-Z^ Canere tvhieU i, 

ormodiiyanim, s etoastandatdsi2ekefer.l!.rf,_ • ' » "«e, mtpand, 

nredncadSv^ ■ ™ fimhe " w,es!, »8«™beperf<»,ned. biases 

P =sQRQ-*p = Mp (7) 



and second images, s denotes (he scale image (which mrm. A . , 
^nera), Q IS the internal camera calibration matrix and * is the ■ 
two camera locations. 10181,0,1 niatnX 

0041] Alternatively, the relationship between th*i™.„ • • 

- • . , P twcenthenm 8 e Projection coordinates p and n' 

>.e., pud locations (x,y) and (x', /), of a stationary world noinf m h 

may be written as: P0B * * tt " 0 COns ^ve images 



(8) 



Where K i rf is the tomography matrix M tha 



(9) 

t maps (ahgns) the first image to the second 



10042] The main task in such image alignment is to determ,™ « 
equation C7),uis dear that given, QM^^^T 

matrix*/ v 4Pen *' eai,d * ,t 18 ^^«Dy straightforward to determine 

(7) also assumes that ^^cea^^^^,^^ 

° f «^^oa m melted embodimenriera 
s.onann.gesy.chmm^ba^andwithe.chimag^ 
F0043, ^^^d^fc..^^^ 

Wis In : "^^^^^--V^nhysomennmber 
of P«el, Posmve translations are in the direction of increasing row or column index- 

rZ OD T; B ^ OSlte ' ^^^^^^onaddsrowsorcoiurnns 
foHowmg approximations may be used to represent such translation, rotation and scaling- 
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(10) 



x -s(xcQsa-ysina) + / A 

y =$(y$ina + xcosa)+t y 
wherein 

s is the scaling (2ooming) factor, 
a is the angle of rotation about the origin; 
tx is the translation in the x direction; and 
t y is the translation m the y direction. 

By introducing new independent variables a, - s cos aa m. a r-..- 

becomes- a ' eqUation °) 



x=a t x-a 2 y + ^ 

(11) 



After during a,. a 2 , t, and *e two images, and I 2 , can be aligned. If these values 
are obtained usmg zoom. pan rad dlt values ^ ^ ^ ^ ^ 

APe and ATe can be determined fox these transformed images using normalized cross- 
correlation. 

board 64 ,n eornmuntcanon with a power supply /1Q board 66. Apower , in e42 connects 
power source 40 to converter 52 in order to provide power to processing device 50 
Processing device 50 receives a raw analog video feed from camera 22 via video line 44 and 
vtdeo hne 45 is used to communicate video images to bead end unit 32. In the illustrated 
embodiment, video lines 44, 45 are coaxial, 75 ohm, 1 Vp-p and include BNC connectors for 
engagement with processing device 50. The video images provided by camera 22 can be 
analog and may conform to either NTSC or PAL standards. When processing device 50 is 

r^r**^* ,,ii " 22 - i - ,ta *^**«» 

to head end umt 32 through analog video line 54, analog circuitry 68, analog video line 70 
and communications plug-in board 72. Board 72 can be a standard communications board 
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capable of handling biphase signals and including a coaxial message integrated circuit 
(COMIC) for allowing two-way communication over video links. 
[0045] Via another analog video line 56, an analog to digital converter 58 receives video 
.mages from camera 22 and converts the analog video signal to a digital video signal After 
the Agftal video signa, is stored m a buffer in the form of SDRAM 60, the digitized video 
uuages are passed to video content analysis digital signal processor (VGA DSP) 62 The 
^W^I^c!^ 460 " a '^°"* n> ^ SSCT ^ 3e ^ above with reference to Figures 2 and 3 is performed 

the v,deo Slg na] is converted to an analog signal. The resulting annotated analog video signa. 
« sent v* analog video lines 76 and 70 to communications plug-in board 72, which then 
sends the signal to head end unit 32 via video line 45. 

I0046J fa the illustrated embodiment, video input to system controller boa* 64 is limited to 
1 Vp-p and rf me video signal exceeds 1 Vp-p it will be clipped to 1 Vp-p. However 
alternative embodiments having a greater or lesser capadty may also be employed wim me 
Present mvention. Processor 62 may be a TriMedia TM-1 300 programmable media 
processor available from PhilUps Electronics North America CorpoxaW At start up 
processor 62 loads a bootloader program. The boot program then copies the VGA apphcation 
code from a memory device such as flash memory 78 to SDRAM 60 for execution. In the 
Hlustrated embodiment, flash memory 78 provides 1 megabyte of memory and SDRAM 60 
proves eight megabytes of memory. Since the application code from flash memory 78 is 
loaded on SDRAM 60 upon start up, SDRAM 60 is left with approximately seven megabytes 



(0047J In the embodiment shown in Figure 4, system controller board 64 is connected to 
communications plug-in board 72 via a biphase digital data bus 1 02, an I2C data bus 1 04 and 
an RS-232 data bus 1 06. System controller board 64 is connected to an RS-232 / RS-485 
compatible transceiver 108 via RS-232 data bus 1 1 0. A line 49. which can be in the form of 

2 RS " 232 dCbU8 ** hU *> C °— icales *■* head end unit 32 to processing device 
50 The Signals on line 49 can include signals that can be modified by processing device 50 
beforebeingsenttocamera22. Such signals may be sent to camera 22 via line 4* fa 
communication with microprocessor 1 ,2. Microprocessor , 12 can operate system controller 
software and may aJso be in communication with VCA components 114. Thus VCA 

components such as VCA DSP 62 can send signals to camera 7-> v ; fi • 

wgns,s 10 camera 22 via micropiocessor 1 12 and 

line 48. 



-15- 



I0048J System controller board 64 may also include a field programmable ^ 
mduding a ma* memory 1 1 8, a character memory 120, and an on screen display (OSD) 
memory 122. Similarly, VGA components 114 may include a mas km emory 124, a character 
me m ory 126, and an on screen display (OSD) memory 128 These components may be used 
to mask various portions of the image displayed on screen 38 or to generate textual displays 
for screen 38. Finally, system controller board 64 can include a parallel data flash memory 
130 for storage of user settings. 
[0049, frtheu,^ 

dev.ce 50 that are input by a human operator are on/off commands, however, even these 
on/off commands may be automated in alternative embodiment Such on/offcommands and 
other serial communications are conveyed via bi-pbase line 46 between head end unit 32 and 
camera 22, and between processing device 50 and camera 22 via line 48. In the illustrated 
embodiment, processing device 50 is provided with a sheet meta, housing and mounted 
proxunate camera 22, however, processing device 50 may also be mounted employing 
alternative methods and at alternative locations. Alternative hardware architecture may also 
beenrployedwrmprocessingdevice50. Such hardware should be capable of™ me 
software and processing at least approximately five frames per second for good results It is 
also noted that by providing processing device 50 with a sheet metal housing its mounting on 
or near a PTZ camera is facilitated and system 20 may thereby provide a stand alone 
embedded platform which does not require a persona, computer-based image stabilization 
system. If desired, however, the present invention may also be employed using a personal 
computer based system. 

fOOSOJ Processing device 50 can perform several functions, including capturing video 
frames acquired by camera 22. identifying a stationary feature m the video frames 
d*ermioing the intended change in the camera FOV based upon signals sent to or 'received 
from camera 22, identifying a stationary feature and determining the actual change in the 
camera FOV, comparing the intended and actual change in the camera FOV to determine the 
magmtude of the image translations resulting from me unintentional motion of the camera 
and selecting display image coordinates to counteract the translations resulting from the 
nmntentional motion of the camera. Processing device 50 may also be used to perform^ 
automated tracking function. For example, processing device 50 may also provide an 
automated tracking system wherein processing device 50 is used to identify moving target 
objects in the FOV of me camera and then generate control signals which adjust the pan tilt 



and zoom settings of the camera to took the target object and maintain the target object 
within the FOV of the camera. As the pan, tilt and zoom settings of the camera are 
automatically adjusted to track the target object the images displayed by the system may be 
stabfiized by utilizing an image stabilization system in accordance with the present invention. 
An example of an automated tracking system that may be employed by system 20 is 
described by Sablak « al. in U.S. Patent Application Serial No. 10/306,509 filed on 
November 27, 2002 entitled "VIDEO TRACKING SYSTEM AND METHOD" the 
disclosure of which is hereby incorporated herein by reference. An image stabilization 
system in accordance with the presenr invention may also be employed to stabilize images 
wherem the camera is being manually adjusted, such as by the manual manipulation of 
joystick 36 or in other applications. 

fOOSl j Figure 5 provides a flow chart which graphically illustrates the general logic of an 
embodiment of the video stabilization algorithm used by system 20 as described above. As 
shown in Figure 5, after turning processing device 50 on, it is initialized at step 1 80 by 
copying the application code from flash memory 78 to SDRAM 60 for execution. Block 182 
represents me remaining memory of SDRAM 60 which is available as a ring buffer for 
storage of video image frames and data associated with each frame received from camera 22 
and stored for processing by processor 62. At decision block 184, processor 62 determines if 
the first flag is true. The first flag is true only when no images from camera 22 have been 
loaded to SDRAM 60 for analysis by processor 62. Thus, when processing device 50 is 
turned on, the first time decision block 1 84 is encountered, the first flag will be true and 
processor 62 will proceed to block 186. Block 186 represents the grabbing of two images by 
processor 62. Processor 62 then proceeds to block 188 where a smoothing filter is applied to 
each nnage. Application of the smoothing filter involves taking two sub-samples of the 
unages. The first sub-sampling step creates a QCIF resolution sub-sample (i.e an image 
having a quarter of the resolution of the original NTSC or PAL OF resolution image) of the 
current /, and * images. The sub-sampling process groups adjacent pixels together to define 
an average value for the grouped pixels. The purpose of the sub-sampling process is to 
reduce the time consumed by the computational processes involve in analyzing the image A 
second sub-sample of the first sub-sample is then taken resulting in images having 1/16 the 
resolution of the original OOP resolution images to thereby further increase the speed of the 
image stabilization process. Block 190 represents the acquisition of the pan, tilt and focal 
length (i.e., zoom) values for each of the images from the camera. (This data represents the 
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intended values and does include the unintentional motion components of the pan and tilt 
values.) Although block 190 is shown at a location foDowing the acquisition of the images 
the pan, tilt and zoom data may be communicated by camera 22 to processing device 50 
simultaneously with the images. 

/0052J Next,block 1 92 represents the calculating of a holography matrix for rotation and 
scahngusmgthepan.tiltandEoomvaluesofthecurreDtimage. Then, in block 194 an 
image transformation is performed by homography to align the second image on to the first 
nnage. (If no pan, tilt or zoom adjustments to the camera have been made during the time 
interval between the acquisition of the two images, no transformation of the images is 
required.) 

10053] At block 196, the image difference of the two aligned images is calculated to 
deternrineifanymotionk P r«entm m eirnages. Thus, after aligning images Ii and I 2 , the 
image difference, A , is calculated in accordance with the following equation: 

<H W, j (I2) 

A histogram of these differences is then calculated. The stationary background of the image 
will typical.y be responsible for generating the largest peak in the histogram. Thus ifthe 
largest peak is centered around zero, the background of the two images is aligned and there 
has not been any unintentional camera movement during the interval between the acquisition 
of the two images. If this is the case, the process returns to decision box 184. If me largest 
peak m the histogram is not centered around zero, this indicates global motion, i.e , the 
background is in motion. This is interpreted as indicating unintentional motion of the camera 
and the process proceeds to box 198. It is also possible that a moving object will be located 
m *■ 3CqUired ™ ages " *«* a object is present in the images, the histogram will 

typically have two peaks associated with it, one corresponding to the background and one 
corresponding to the moving object If this is the case, the background of the image is 
assumed to take up more area of the image than the moving object and the largest peak value 
is used to make the above determination. In other words, the histogram is evaluated to 
determine whether the largest peak of the histogram is centered on zero, indicating the: 
absence of unintentional camera movement, or whether the largest peak indicates global 
movement, signifying the presence of unintentional camera movement For example, Figuxe 
6 schematically depicts a histogram of image differences wherein the primary peak is 
centered on zero, indicating no unintentional camera movement, and also including a 
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f0054] If unintentional camera movement is detected and the 

~ « . r , "ciccrea and the system proceeds to block 1 9* 

determining the value of AT E and AP E . -responds to 

^^^^^^^^^^^^^^^ 

provide a stabilized video image. £ mCTeby 

The process then proceeds to block 192. * 
^ rao ^ maIwispotaedbyprocess|nedevi . ce5o ^ 

"age. ^ 22. ^ „ - 

«* « . *— m»1 . Mock IN rm «. chm of Fign „ ^ y 

forth. ™ ^ re0llire adjllsMem correct for the mtmm J 
of*. ^ ^ ^ ^ ^ ^ ^ 
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Sethng the location of the display boundary i„ these additional unanalyzed images can be 
done m different manners. For example, the display boundary used with these additional 
rmages utilize the adjusted boundary position, eg., display boundary 1 00, that resulted from 
the analysts of the last two grabbed images could be employed with the unanalyzed images 
untri the stabilization process positively deterruined that the display boundary should be" 
readjusted. Alternatively, after altering the display boundary for the one image which was 
specially analyzed and determined to have been influenced by unintentional camera 
movement, me display boundary could be immediately returned to hs centered location in the 
CCD dasplay. In other words, the display boundary would remain at its centered location for 
all miages except for those specific images which have been grabbed, analyzed and 
determined to have been affected by unintentional camera movement and for which an 
altered display image boundary is specifically determined. It is also possible for the display 
boundary to be returned to the centered location m die CCD image boundary in incremental 
amounts, over several sequentially acquired images, after the display boundary has been 
adjusted to account for unintentional camera movement. 

I00S9] In most surveillance camera applications, it is anticipated that the types of vibrations 
and other external forces acting on the camera will cause unintentional camera movements 
that have a very brief deviation from the intended position with possible oscillations about an 
equilibrium position corresponding to the intended position of the camera. In this type of 
situation, having the display boundary rem™ immediately to the centered location can be 
beneficial. Returning to the centered position may not only may reflect the underiying 
unintentional camera motion and thus facilitate the display of a stabilized image, it also 
prevents migration of the display boundaries to the edge of the CCD display image which 
could limit the effectiveness of the image stabilization process and could result from a 
process which does not return the display image boundaries to a centered location in the CCD 
image. 

(0060] While this invention has been described as having an exemplary design, the present 
mvennon maybe further modified within the spirit and scope of this disclosure. This 
application is therefore intended to cover any variations, uses, or adaptations of the invention 
using its general principles. 
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WHAT IS CLAIMED IS: 

1 . A video image stabilization system comprising: 
a camera including an image capturing device configuted to capture a video i 
said camera having a selectively adjustable field of view; and 

at least one processing device operably coupled to said system wherein said 
processing device receives signals indicative of the field of view ofsaid camera and images 
captured by said camera, said processing device sequentially grabbing a first one ofsaid 
images and a second one ofsaid images, said processing device determining a stabilizing 
adjustment for the video image as a function of an intended change in the field of view of 
said camera during an interval between the capture ofsaid first image and said second image 
— upon said signals indicative of the field of view and an analysis ofsaid first and second 



2. The video image stabilization system of claim 1 wherein a display 
portion of each image captured by said camera is selected for display, said selected display 
portion of each image comprising less man the entire captured image and wherein said 
stabilizing adjustment determined by said processing device comprises adjusting a relative 
location ofsaid selected display portion within said captured image. 

3. The video image stabilization system of claim 1 wherein said camera 
has a selectively adjustable pan position, a selectively adjustable tilt position and a selectively 
adjustable focal length. 

4. The video image stabilization system of claim 3 wherein at least one of 
the pan position and the tilt position ofsaid camera is intentionally adjusted between the 
capture of said first and second images. 

5. The video image stabilization system of claim 3 wherein the focal 
length ofsaid camera is intentionally adjusted between the capture ofsaid first and second 
images. 

6. The video image stabilization system of claim 5 wherein said analysis 
ofsaid first and second images includes transforming one ofsaid first and second images 
wherein the scale of the one image is adjusted. 

7. The video image stabilization system of claim 1 wherein said analysis 
ofsaid first and second images includes aligning one ofsaid first and second images with the 
other ofsaid first and second images based upon said signals indicative of the field of view 
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and determining if said aligned images indicate the occurrence of unintentional camera 
movement 

8. The video image stabilization system of claim 7 wherein determining if 
said aligned images indicate the occurrence of unintentional camera movement comprises 
determining an image difference of said aligned images. 

9. The video image stabilization system of claim 8 further comprising 
generating a histogram of said image difference of said aligned images and determimng 
whether unintentional camera movement has occurred based upon identifying a maximum 
peak m said histogram and a location of said maximum peak. 

10. The video image stabilization system of claim 9 wherein when said 
maximum peak is not substantially centered on zero, unintentional morion is determined to 
have occurred and wherein determination of said stabilizing adjustment includes identiiying 
adjustments that minimize said image difference. 

11. The video image stabilization system of claim 1 wherein said signals 
indicative of the field of view comprise pan, tilt and focal length settings of said camera for 
each captured image, said signals being communicated to said processing device from said 
camera on an image-synchronized basis 

12. The video image stabilization system of claim 1 wherein said camera is ' 
moveably mounted on a stationary support. 

13. A video image stabilization system comprising: 

a video camera including an image-capturing device configured to capture images, 
each captured image associated with a field of view, said camera having at least one 
selectively adjustable parameter wherein adjustment of said at least one camera parameter 
varies the field of view of said camera; 

a display device configured to display a selected portion of images captured by said 
camera; and 

a processing device operably coupled to said camera and to said display device 
wherein said processing device receives signals indicative of said at least one camera 
parameter and images captured by said camera, said processing device being operable to 
compensate for inadvertent movement of said video camera by adjusting the selected portion 
of the images displayed by said display device based upon the signals indicative of said at 
least one camera parameter and an analysis of a current captured image and a previously 
captured image. 



14. The video image stabilization system of claim 1 wherein said signal 
indicative of at least one camera parameter includes a signal indicative of pan, tilt and zoom 
settings. 

1 5. The video image stabilization system of claim 1 wherein the selected 
portion of the video image comprises a central portion of the video image. 

1 6. The video image stabilization system of claim 1 wherein the selected 
portion of the video image includes approximately between 60% and 90% of the video 
image. 

1 7. A method of stabilizing a video image, said method comprising: 
capturing a plurality of images with a video camera, the video camera being mounted 

on a stationary support and having an adjustable field of view; 
adjusting the field of view of the camera, 

grabbing a first image and a subsequent second image captured by the video camera; 
displaying a selected display portion of each of the first and second images, the 
selected display portion of each image being less than the entire captured image; and 

determining a stabilizing adjustment for the second image as a fimcrion of an intended 
change in the field of view of the camera during an interval between the capture of the first 
image and the capture of the second image and an analysis of the first and second images 
wherein the stabilizing adjustment involves adjusting a relative location of the selected 
display portion within the second image. 

18. The method of claim 1 7 wherein the camera defines a pan position and 
a tilt position and at least one of the pan position and the tilt position of the camera is 
intentionally adjusted between the capture of said first and second images. 

19. The method of claim 17 wherein a focal length of the camera is 
intentionally adjusted between the capture of said first and second images. 

20. The method of claim 19 wherein the analysis of the first and second 
images includes transforming one of the first and second images wherein the scale of the one 
image is adjusted. 

21. The method of claim 17 wherein the analysis of the first and second 
images includes aligning one of the first and second images with the oiher of the first and 
second images based upon signals indicative of the field of view received from a system in 
operative communication with the camera and determining if the aligned images indicate the 
occurrence of unintentional camera movement. 
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22. The method of claim 21 wherein determining if the aligned images 
indicate the occurrence of unintentional camera movement comprises determining an image 
difference of the aligned images. 

23. The method of claim 22 further comprising generating a histogram of 
the image difference of the aligned images and determining whether umntentional camera 
movement has occurred based upon identifying a maximum peak m the histogram and a 
location of the maximum peak. 

24. The method of claim 23 wherein when ihe maximum peak is not 
substantially centered on zero, unintentional motion is determined to have occurred and 
wherein determination of said stabilizing adjustment includes identifying adjustments that 
minimize the image difference. 

25. A method of stabilizing a video image; said method comprising: 
intentionally varying a field of view of a video camera by use of control signals; 
capturing a plurality of images with the camera, each of the images being associated 

with a field of view; 

displaying a selected portion of each of the plurality of images; and 

adjusting the selected portion that is displayed for at least one of the plurality of 

images, said adjusting step compensating for inadvertent movement of the video camera and 

being based upon an intended field of view of the camera and an analysis of the at least one 

of the plurality of images and a second one of the images. 
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